Dense Feature Fusion for Online Mutual Knowledge Distillation

Abstract Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities, Ignoring middle-level supervision, resulting in low efficiency training many models. Even if some have joined way is to define characteristic loss, effect general. We propose a new method of through fusion features between teacher academic network enter supervision. The specific fuse network, establish an auxiliary branch process information, so that feature can effectively strengthen interaction network. At same time, we added normalized integrated output, our reached SOTA KD. done lot experiments on cifar-10, cifar-100 ImageNet datasets, proved this more effective than other performance sub classifier, as well generating meaningful maps.